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FALL 2021 VOLUME OVERVIEW 


Completing a degree that leads to gainful 
employment at a reasonable cost is a key desired 
outcome of higher education. Many stakeholders 
want reliable job placement and return-on- 


investment information to understand how well 


institutions achieve this outcome. Prospective 


students want help making decisions about whether 


+ 


to attend college, where to attend, and what 


+ 


major to select. College gradua 


+ + 


tes are interested 


in earnings data for their degrees so that they 


are better prepared for salary negotiations. State 


policymakers are concerned with student loan 


default rates and the shares of graduates choosing 


to leave the state. Job placement and college costs 


are the topics covered in the current issue of The A/R 


Professional File. 


In the article Using State Workforce Data to Report 
Graduate Outcomes, Matt Bryant shares the process 
and methodology used to measure employment 
outcomes of graduates of a public university and 
compares survey data with state wage records. 

This article is an excellent source of information for 
institutions seeking innovative methods for studying 


graduates’ earnings and job placements. 


In their article How Noncredit Enrollments Distort 
Community College IPEDS Data: An Eight-State Study, 
Richard M. Romano and Mark M. D'Amico discuss 
implications of excluding noncredit enrollments 
when calculating IPEDS expenditures per student 
full-time equivalent (FTE). As IR professionals, we 
are frequently asked to benchmark our institutions 
against peers, and a commonly used IPEDS metric 


for benchmarking college costs is expenditures per 
student FTE. The results of this study help us better 


understand the limitations of this measure and how 


excluding noncredit enrollments from expenditures 
per FTE calculations has differential effects across 


institution types. 
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Abstract 


Survey methodology is the dominant approach 
among universities in the United States for reporting 
employment outcomes for recent graduates. 
However, past studies have shown that survey 
methodology may yield upwardly biased results, 
which can result in overreporting of employment 
rates and salary outcomes. This case study describes 
the development and application of an alternative 
reporting methodology, by which state wage records 
are analyzed to determine employment and salary 
outcomes for recent graduates. Findings at Western 
Washington University suggest the significant 
sample sizes that can be achieved using wage record 
methodology may provide a more reliable option 
than survey methodology for accurately reporting 


graduate outcomes. 


Keywords: graduate outcomes, graduate 
employment, earnings reporting, wage records data 
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Introduction: Why Not Survey? 


For decades, universities in the United States 

have relied on survey methodology to report 
employment outcomes for recent graduates. Survey 
methodology has been recognized as a best practice 
for reporting graduate outcomes since the late 
1970s (Pace, 1979). More than 360 universities use 
survey methodology every year to report graduate 
outcomes to the National Association of Colleges 
and Employers (NACE), an organization considered 
to be the national authority on graduate outcomes 
reporting (NACE, 2019a). Western Washington 
University (WWU), the sample institution for this 
case study, has used survey methodology to report 
graduate outcomes through its annual Employment 
Status Report since 1990. 


Despite the history and prevalence of survey 
methodology, survey results might not accurately 
reflect graduate outcomes due to nonresponse 
biases. Meta-analysis of survey research shows 

that “large nonresponse biases can happen in 
surveys,” and indeed such biases are often present 
(Groves & Peytcheva, 2008, p. 183). In particular, 
analysis of graduate follow-up surveys find that 
successful graduates are more likely to respond, 
yielding upwardly biased results and thus negatively 
impacting external validity (Dey, 1997; Grosset, 
1994). This sort of bias has been observed across 
survey elicitation methods; while it can sometimes 
be corrected through postsurvey weighting 
techniques, such adjustments require auxiliary 
variables that are typically beyond the scope of 
such instruments (Dey, 1997; Groves, 2006; Hudson 
et al., 2006). 


Bafflingly, NACE addresses neither nonresponse 
biases nor data validity in its standards and 


protocols for graduate outcomes surveying. NACE 
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does recommend a minimum response rate (using 
the term “knowledge rate”) of 65% (NACE, 2019b). 
However, while “good survey practices dictate 
striving for a high response rate as an indicator of 
the quality of all survey estimates” (Groves, 2006, 

p. 670), itis also the case that “higher response 
rates do not necessarily reduce nonresponse 

bias" (Groves, 2006, p. 663). Nevertheless, “overall 
response rate is one guide to the representativeness 
of the sample respondents” and a minimum survey 
response rate of 50% is considered “adequate for 


analysis and reporting" (Babbie, 2001, p. 256). 


Unfortunately, WWU saw the knowledge rate for 

its employment status reports gradually decline 
from more than 60% in the 1990s to around 40% in 
the 2010s, which is consistent with national trends 
in survey research (Groves, 2006; WWU 1990, 
2016a). Furthermore, the knowledge rate reported 
throughout the 2010s was not the actual survey 
response rate, but rather the percent of graduates 
for whom any data were collected, either by survey 
or by a supplementary process. This supplementary 
process (which by 2014 was included as a suggestion 
in the NACE protocols) was to search the LinkedIn 
website for employment records for graduates, and 
to include those with employment records on the 
website in the report sample (NACE, 2019b). 


Regrettably, this modified survey methodology 
resulted in a significant percentage of the sample 
being selected based on the primary outcome 
variable of the study. Since LinkedIn data can be 
sampled only for those with recent employment 
records, those data can capture only positive 
employment outcomes. This sampling methodology 
represents a systematic form of selection bias that 
positively skews results and negatively impacts 
internal validity (Mare & Winship, 1992). Additionally, 


this process provided little information beyond 
when and where an individual was employed; 
consequently, other measures in the employment 
status reports were presented as representative of 
the population, despite being based on only a very 
small number of survey responses. For example, the 
average Salary reported for 2015-2016 bachelor's 
degree graduates (a population of 3,280 individuals 
with a reported sample of 45%) was based on 

only 100 survey responses (since salary data are 
not available on LinkedIn, and the salary item was 
optional on the survey instrument), and therefore 
represented only 3% of the overall population 
(WWU, 2017). 


Finally, by 2016 it had become apparent that the 
employment status reports for WWU were in 
conflict with data from other sources. Since 2012 
the university alumni survey (administered 1 to 2 
years after graduation) had yielded employment 
rates 5%-9% lower than those presented in the 
employment status reports (administered 9 to 12 
months after graduation), and did so with more- 
robust methodology and higher (actual) response 
rates (WWU, 2016b). Clearly, WWU needed better 
methodology in order to report outcomes more 
accurately’ for recent graduates. This case study 
describes the process of identifying, developing, 
and implementing a new methodology for 
reporting employment outcomes based on state 
wage records, and presents initial findings and 


considerations for further research. 


ALTERNATIVES: 
AVAILABLE DATA 
SOURCES 


When initial efforts began in 2017 to implement 
new methodology for reporting graduate outcomes 
at WWU, the author was unable to identify a 

single public university in the United States that 
was reporting employment rates using any form 

of alternative methodology.? Without exception, 
every university in the Pacific Northwest region and 
every peer institution in the country was reporting 
graduate outcomes using survey methodology (of 
various levels of modification) according to the 
NACE protocols. 


Graduate outcomes consist of two primary 
measures: percentage of graduates employed, and 
percentage of graduates seeking further education 
(within 6 months of graduation, according to NACE). 
For the latter measure, subsequent enrollment 
records for graduates were already available to 
WWU through the National Student Clearinghouse; 
this rate could be calculated with a high degree 


£ 


of accuracy. However, there was no comparable 
national source for employment records available 
n 2017, and there is still no such resource available 


in 2021. 


U 


ublicly available employment data exist in three 
primary forms: as national census data (based on 
survey responses), as national social security data 
(based on tax records), and as state wage record 
data (collected directly from employers). However, 
the Higher Education Opportunity Act of 2008 


1. To be clear, | describe here the former practices at WWU not to call into question the work of the fine professionals who preceded me, but because those 
professionals were simply observing best practices advocated by a leading national organization, and because similar practices are still in use presently at many 


other institutions. 


2. The University of Texas began providing graduate earnings data based on wage records in 2014; however, these data were not used to determine 
employment rates and reflected earnings starting only 1 year after graduation, and not during the critical 6-month postgraduation time period typically 


emphasized in graduate outcomes reporting. 
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banned the linking of national data to student-level 
record information, making it impossible to use 
federal data sources at the institutional level for 
outcomes reporting. This left wage record data, 
which were already being used in Washington State 


to report average earnings for graduates 1 or more 
years after graduation (Education and Research 
Data Center [ERDC], 201 7). 


However, although wage records “account for 


the vast majority of workers in a state” and are 
“considered to be one of the most accurate data 
sources,” these records were presumed to provide 
insufficient data for calculating employment rates 
(Troutman and Shedd, 2016, p. 89). In fact, in the 
article that inspired this case study, this argument 
was presented on the basis that wage data are 
limited to workers employed in a given state, and 
therefore “without data on employment out-of- 
state, institutions would be underestimating, to 
varying levels of degree, the true employment rate 
of graduates” (Troutman and Shedd, 2016, p. 89). 
There was also the limitation that wage data cannot 
account for all types of employment, because the 
records exclude federal and self-employed workers, 
and therefore cannot provide outcomes even for 
all graduates employed in-state. Nonetheless, while 
the apparent shortcomings of using wage records to 
report graduate outcomes proved to be challenges, 
it was determined these limitations could be 
accounted for with appropriate methodology (see 


Methods section, this article). 


PROCESS: ACCESSING 
DATA 


In many states, individual-level wage records 

are available to public institutions for evaluative 
purposes through state agencies, but these 
institutions can access the records only once an 
appropriate data-sharing agreement has been 
established. The specific agency and process for 
establishing such an agreement will vary by state, 
and those interested in obtaining wage records 
should begin the process by contacting their state 
labor office.* For this case study, a data-sharing 
agreement was established with the Employment 
Security Department (ESD) of Washington State, thus 
providing access to wage records (including wages 
earned, hours worked, and employer details) for 
WWU graduates. 


Worth noting here is the timeline of this process. 
Because graduate employment outcomes are 
measured 6 months after graduation,’ it is 
necessary to wait two fiscal quarters after the last 
members of the cohort have graduated in order 

to capture outcomes for those graduates who 
gained employment during the second quarter. 
Furthermore, it is necessary to wait an additional 
fiscal quarter to capture a full quarter of earnings 
for those graduates who gained employment during 
the second quarter, since records of employment 
gained during the second quarter will not include a 


full quarter of wages for estimating annual earnings. 


3. The data-sharing agreement for this case study required nearly 6 months to be finalized; those interested in establishing such an agreement may wish to 


initiate this process as soon as possible to ensure timely access to data. 


4. It is also worth noting that wage record methodology provides an advantage over survey methodology in observing this reporting timeline. NACE recommends 
surveying each graduating class the following December, which results in capturing outcomes 15 months after graduation for those cohort members who 
graduated in the summer quarter, and outcomes 6 months after graduation for those cohort members who graduated in the spring quarter (for institutions 
using quarterly scheduling). Wage record methodology allows for effectively analyzing results for an entire cohort exactly 6 months after graduation, regardless 


of the term of completion for each individual. 
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In the state of Washington wage records become 
available approximately 8 weeks after the end of 
the fiscal quarter. Therefore, for a full analysis of 
outcomes for an annual class of graduates (by the 
methodology described in the Methods section), 

it is necessary to wait three fiscal quarters plus an 
additional 8 weeks, or approximately 11 months, 

for data to become available. Likewise, to capture 
educational outcomes for individuals 6 months after 
graduation, it is necessary to wait for enrollment 
records to become available for the third quarter 
after graduation to capture outcomes for those 
graduates enrolled in programs that commenced at 
the end of the 6-month time period. (The preceding 
is based on the assumption that intent to enroll was 
established during that time period, and to account 
for the varying academic calendars and program 


start dates across the myriad higher education 


institutions in the United States.)? It is worth noting 
that this timeline does not yield outcomes data 

by the deadlines set by NACE to be included in its 
annual First-Destination Survey. This timeline can be 
considered a limitation of the methodology applied 
for this case study, or a weakness of the current 
methodology used by NACE for reporting graduate 
outcomes, the timeline of which is simply not long 
enough to allow for using actual employment and 


enrollment records to verify or report outcomes. 


To request wage records, it is necessary to 
provide identifying data for individuals. To analyze 
annual graduate employment outcomes, it is 
necessary to provide identifying data for all 
individuals in the defined graduating class. For the 


methodology described in the Methods section, the 
group of all graduates in a given class represents the 


study population. 


State agencies responsible for processing wage 
record requests will likely require social security 
numbers to identify individuals. For the purpose 

of this case study, student university identification 
numbers were also provided, allowing for social 
security numbers to be removed from the returned 
records, while maintaining the ability to identify 
individuals. Those university identification numbers 
were then matched to institutional records for each 
individual, including degree program, demographic 
data, and permanent address. Finally, these data 
were matched with enrollment records for those 
individuals, requested through the National Student 
Clearinghouse, to allow for comprehensive analysis 


of outcomes for the study population. 


METHODS: CALCULATING 
OUTCOMES 


Once enrollment and wage records have been 
obtained for a graduating class, data should be 
cleaned, matched, and sorted for analysis. Troutman 
and Shedd provide excellent guidance on data 
cleaning and preparation for analysis, including a 
decision tree for processing inconsistencies in wage 
records, which is a tedious but necessary task “to 
ensure data are accurate and reliable” (Troutman 
and Shedd, 2016, p. 23). Ata minimum, this process 
requires removing unidentified and duplicate 
records, appropriately sorting instances of multiple 
records (for graduates with multiple employers), and 
identifying any outlier records to be excluded from 
the sample dataset.® A variety of software programs 
of varying degrees of sophistication can simplify this 
process; for the purposes of this case study, the 


use of Microsoft Access with simple queries to join, 


5. At WWU the academic year ends in June. As a result, enrollment records for each annual cohort are requested from the National Student Clearinghouse the 
following March, and wage records from the ESD the following May. These records are used to calculate outcomes for an annual report of graduates from the 


previous academic year; the report is published each June. 


6. The ERDC of Washington State also provides useful guidance for linking and cleaning data in its Employment Data Handbook (ERDC, 2012). Note that part of the 
data-cleaning process at WWU has included removing from the sample any individual records reflecting less than 1 hour of work at minimum wage in order to 
avoid skewing aggregate outcomes with isolated earnings unrepresentative of continuing employment. 
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compare, and filter records proved to be sufficient on accurately defining the measures to be analyzed. 
for data preparation. As mentioned above, the population for each report 
consists of all individuals in the graduating class, and 


Once data have been properly prepared, definitions are provided in Table 1 for each measure 
calculations for graduate employment outcomes are of that population. 


not particularly sophisticated, and primarily depend 


Table 1. Measure Definitions 


Sample Population individuals with permanent addresses in the state 

Seeking Further Education Sample individuals with enrollment records within three quarters of 
graduation 

Employed Sample individuals not enrolled in further education with positive wage 


records in the second quarter after graduation 


Earnings Annualized earnings based on third quarter wage records for individuals 
considered “Employed” 


Other Sample individuals who do not meet criteria above for “Seeking Further 
Education” or “Employed” 


Calculating these measures is a fairly straightforward 3| Wage records are then analyzed for the second 


process: quarter after graduation for the remaining 
individuals in the sample (those not “Seeking 
1| The sample is selected based on permanent Further Education”). Those with positive wage 
address. If, at the time of analysis, an individual records in the second quarter are considered to 
has an in-state permanent address in the be “Employed.” 


tudent informati tem, that individual i ’ 
eon ae a nc 4| Wage records are then analyzed for the third 


j j a. 
NGUAEC A EHeSamPE: quarter after graduation for each individual 


2| National Student Clearinghouse enrollment considered to be “Employed,” with third quarter 
records are then analyzed for the second and wages for each individual quadrupled to 
third quarters after graduation for each individual estimate annual earnings. 


in the sample. Those with enrollment records in 
the second or third quarter are considered to be 
“Seeking Further Education,” and wage records 
are not analyzed for those individuals. 


7. This is a critical component of the methodology for this study. The permanent address field in the student information system at WWU is updated quarterly 
based on United States Postal Service records and alumni outreach efforts and is considered to be a reliable indicator of a graduate's current residence. If a 
high percentage of graduates reside in-state, then limiting the sample to those individuals represents a workaround to the issue of not capturing out-of-state 
employment. With approximately 90% of WWU graduates residing in-state, sample sizes have provided confidence in assuming outcomes for in-state graduates 
are also generally representative of graduates who reside out-of-state. This suggestion is supported by the fact that Washington has the second-highest median 
wage of any state and also has a statistically average employment rate. Even if all graduates who left Washington ended up moving to Massachusetts, which is 
the state with the highest median wage and among the highest employment rates, employment and average earnings for the population would increase by less 
than 0.5% overall (StatsAmerica, 2020). However, this methodology may not be effective for institutions with graduates who are more geographically dispersed, 
although it could be possible to obtain wage records from neighboring states to account for broader geographical dispersion (Troutman and Shedd, 2016). 
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5| The remaining individuals in the sample 
(those not “Seeking Further Education” or not 
“Employed") are considered to be “Other,” a 
category that accounts for graduates who are 
still seeking employment or who are engaged 
in other activities (nontraditional employment, 
parenting, traveling, volunteering, etc.). 


Once these calculations have been made, some 
adjustments are necessary to determine accurate 
outcomes rates. The issue of wage records 
capturing only in-state employment is addressed 
by limiting the sample to graduates living in-state 
(see footnote 7). In order to address the issue 

of wage records capturing only certain types of 
employment, however, it is necessary to adjust the 
employment rate to reflect unrepresented types. 


Wage data “cover approximately 90 percent of the 
workforce, including private businesses, state and 
local governments, some nonprofit organizations, 
and Indian Tribes” (Mullin, 2012, p. 76). This results 
in two primary unrepresented types of employment: 
federal employment and self-employment.’ In order 
to account for these types of employment, the 
sample employment rates for this case study have 
been adjusted based on the federal employment 
(2.2% [Office of Personnel Management (OPM), 
2017]) and self-employment rates (10.5% [Hipple & 
Hammond, 2016]) for Washington State. Applying 
these rates as ratios, a total proportional increase 
of 14.5% was made to the sample employment 
rate,’ with corresponding decreases to the rates of 
students categorized as “Other,” to determine the 
overall employment rate for each cohort. 


8. The ESD of Washington State publishes a useful guide on types of nonfederal occupations that are not represented in unemployment insurance wage data. A 
thorough review reveals a variety of occupations that can be categorized as either nontraditional work (elected, religious, small-scale [less than $1,000/quarter], 
inmate/patient/student, unpaid) or self-employment work (ESD n.d.). For the purposes of this study, nontraditional employment is assumed to be minimal and to 
be reflected in the “Other” category; federal and self-employment are accounted for by the adjustment described in footnote 9. 


9. For this adjustment, the federal and self-employment rates noted above are used to determine a ratio for federal/self-employed workers to other worker 
types in the state, a value of .145. That value is then applied to the rate of other worker types observed in the sample, to account for unrepresented types of 


workers in calculating the overall employment rate. 
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FINDINGS: ANALYZING 
OUTCOMES 


In 2018 WWU began using the wage records 
methodology described above to report outcomes 
for the 2016-2017 graduating cohort. However, 
WWU also requested records for the 2014-2015 


Table 2. Outcome Comparisons 


and 2015-2016 graduating cohorts, for whom 
outcomes were previously reported using modified 
survey methodology, in order to compare outcomes 
by methodology for the same cohorts. Cumulative 
outcomes by methodology type for each of those 
cohorts are presented in Table 2. 


Population 3,174 3,174 3,308 3,308 
Sample 42.7% 93.4% 45.2% 92.3% 
Employed 82.7% Vse296 83.1% 75.6% 
Further Education | 11.9% 12.0% 13.2% 12.9% 
Other 5.4% 12.8% 3.7% 11.5% 
Salaries Reported 296 1,945 100 ZFOj 
Average Salary $38,732 $29,782 $43,570 $30,083 


Source: WWU, 2016a, 2017. 


Basic analysis comparing the different outcomes sets 
in Table 2 indicates that, for each cohort, modified 
survey methodology produced an approximately 
10% higher employment rate (for an overall rate 
increase of 7.5 percentage points) and a 30%-45% 
higher average salary measure, based on a 50% 
smaller sample than wage record methodology. 
These comparisons provide some indication of the 
extent to which the modified survey methodology 
used previously may have resulted in overreporting 
of graduate outcomes. Furthermore, the significant 
sample sizes that can be achieved using wage 
record methodology suggest it may be the most 
reliable option for accurately reporting graduate 


outcomes. For this case study, differences in sample 
sizes between modified survey and wage record 
methodology are especially apparent with regard to 
the average salary measure. Wage records provided 
salary data for more than 60% of the population, 
whereas Survey responses previously provided 
salary data for less than 10% of the population. At 
WWU, where regional and demographic factors 
allow for sampling sizes exceeding 90% of the 
population, wage record methodology has proven 
to provide more-consistent and more-reliable data 
for a far greater number of graduates and has 
consequently become the primary approach used at 
the university for reporting graduate outcomes."° 


10. WWU has since implemented a graduate outcomes report dashboard summarizing wage record data for cohorts dating back to 2006-2007 (using a 
slightly modified version of the methodology described in this article), and no longer publishes past employment status reports. The graduate outcomes report 


dashboard can be viewed at WWU (2021). 
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DISCUSSION: 
CONSIDERING 
IMPLICATIONS 


While the wage record methodology described in 
this article was designed to minimize confounding 
factors, it does rely on some imperfect assumptions: 
that most graduates are working in the state in 
which they reside, that outcomes for in-state 
graduates are generalizable to out-of-state 
graduates, and that graduates acquire federal 
employment and self-employment at the same 
rates as the state workforce. There are also 

some additional limitations to using wage record 
methodology to report graduate outcomes. As 
previously mentioned, the timeline for requesting 
wage records does not allow for the resulting 
outcomes to be submitted for inclusion in NACE's 
First-Destination Survey, an annual national 
publication. More significantly, wage records do 
not include job titles, a key piece of outcomes 
information often requested by prospective 


students, which is critical for determining a rate for 


field-related employment. While it is not possible 

to determine whether employment reflected in 
wage records is field-related, job titles for individual 
graduates can be reported more reliably using 
survey methodology. At WWU, sample job titles for 
recent graduates are reported separately based on 
responses collected for the university alumni survey. 
Finally, wage records might not allow for reporting 
outcomes for very small programs or majors without 
the risk of aggregate data becoming individually 
identifiable. At WWU, employment outcomes are 


suppressed for programs or majors with four or 


fewer members in the sample for this reason. 


This case study presents one possible methodology 


for using wage records to report graduate 
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outcomes. Depending on the institution, other 
methodologies for using wage records could be 
more appropriate. In particular, institutions with 
graduates who are more geographically dispersed 
might need to obtain data from neighboring states 
to accurately report employment outcomes. In 
2018 a partnership was announced between the 
University of Texas System and the United States 
Census Bureau's Longitudinal Employer-Household 
Dynamics (LEHD) program to report national 
aggregated data of graduate earnings by institution 


and major. This reporting was accomplished 
through a “powerful but ‘imperfect’ workaround to 
the current ban on a federal database that would 
link student-level educational data to national 
employment data, which was forbidden by the 2008 
Higher Education Act” (Bauer-Wolf, 2018, para. 2). 
Partnering with the LEHD program might provide 
public institutions with access to national wage 
record data, but partnerships must be established at 
the state level, and only nine states currently provide 
data for a majority of graduates (United States 
Census Bureau, 2021). 


Additionally, for those institutions without access 
to wage records data, this case study may 

provide some best practices in improving survey 
methodology to report graduate outcomes more 
accurately. One of these best practices may be 

to avoid the modified process for data collection 
advocated by NACE (using LinkedIn profiles), which 
results in selection bias, positively skews results, and 
negatively impacts internal validity. Another best 
practice may be to collect identifying information 
and auxiliary variables for survey respondents, 

in order to correct nonresponse bias through 
postsurvey weighting. It might also be useful to 
require critical items (e.g., salary) to maintain 
strength of sample size, as well as to include items 


beyond economic outcomes (e.g., satisfaction 


12 


questions) to encourage engagement from those 


with less-favorable outcomes. 


Finally, it is worth mentioning that there are other 
potentially unexplored uses for wage records 

data that could prove meaningful to institutional 
research efforts. Wage records data, including hours 
worked, wages earned, and employer industry, 

can be used to analyze employment outcomes by 
demographics, to measure the impacts of specific 
employment support interventions, and even to 
analyze the impacts of employment for students 
who are currently enrolled. Ultimately, wage records 
represent a promising data source for use by higher 
education administrators to increase accountability 
and transparency, and to inform efforts to improve 


graduate outcomes. 
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